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ABSTRACT 



The Association of Research Libraries (ARL) has collected 



descriptive data from its members for the better part of the 20th century. 
This data shows that as libraries 1 environments change to reflect increased 
interest in accountability and institutional outcomes. To account for this, 
an ARL New Measures Initiative has been established to develop different and 
innovative ways for library staff to describe the library's contributions to 
various institutions. The largest of the New Measures Initiatives to date is 
LibQUAL, a research and development project undertaken by ARL in 
collaboration with Texas A&M University to define and measure library service 
quality across institutions and to create useful quality-assessment tools for 
local planning, such as the evaluation of a library's collections-related 
services from the user's point of view. Another initiative, the E-metrics 
project, is an effort to explore the feasibility of collecting data on the 
usage of electronic resources. These new measures will assist libraries in 
moving away from data that just describe a library's input and output, and 
moving toward data than can help a library measure its own performance level 
over a period of time, both to benchmark with similar libraries and to 
improve that library's own operations. (Contains 20 references.) (MES) 
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Abstract: v 

The Association of Research Libraries (ARL) has collected descriptive data from its members for the 
better part of the 20 th century. As the libraries' environments change to one of increased interest in 
accountability and institutional outcomes , an ARL New Measures Initiative has been established to 
develop different and innovative ways for libraries to describe their contributions to their institutions. 
These new measures will assist libraries to move away from data that just describe a library ’s inputs 
and outputs to data and programs that can help libraries measure their performance over time both to 
benchmark with peers and to improve their own operations. 
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Introduction 

The Association of Research Libraries (ARL) is a not-for-profit membership organization comprising the 
leading research libraries in the United States and Canada. Its mission is to shape and influence forces 
affecting the future of research libraries in the process of scholarly communication. Although ARL libraries 
are a relatively small subset of the research libraries in the United States and Canada, they account for a large 
portion of academic library resources in terms of assets, budgets, and the number of users they serve. 
Statistics have been a part of ARL’s programs since the association began, and have had a significant impact 
on the development and use of library statistics in the United States and Canada throughout the 20 th century. 

Development of ARL Descriptive Statistics 

Since 1961-62, ARL has collected data annually from and published statistical data for its members that 
describe their collections, expenditures, and staffing. Prior to 1961-62, annual statistics for university 
libraries were collected by James Gerould, who was first at the University of Minnesota and later at 
Princeton University. 1 These data, covering the years 1907-08 through 1961-62, are now called the Gerould 
statistics. 11 The whole data series from 1908 to the present represents the oldest and most comprehensive 
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continuing library statistical series in North America. Machine readable datafiles dating from 1908 are 
available on the ARL ftp server 111 and an interactive interface has been developed and is supported by the 
staff at the Geospatial and Statistical Data Center at the University of Virginia. lv 

The ARL descriptive measures dataset has remained relatively unchanged throughout the years. The data 
variables used in the 1960s to the 1980s, with only minor modifications and additions, were those first 
established by Gerould. Over the past decade, the set of data gathered was expanded to include more 
categories for public service activities in addition to the traditional categories of library characteristics, 
collections, personnel, expenditures, and university data. Except for a few services, the library variables still 
concern the inputs of on-site collections, staff, and expenditures/ 

ARL statistics data have been most useful at the aggregate, national level for the trend information they 
provide about changes in services (interlibrary borrowing, group presentations, circulation, reference 
transactions) in the last 10 years vl and monograph and serial unit costs; monograph and serial expenditures, 
and monograph and serials purchased since 1986.™ These data and their accompanying charts are used by 
the Association and its members to document the decline in ownership due to the high cost of materials. The 
value of reliable trend data can not be overstated. 

However valuable the ARL data were for describing the traditional characteristics of research libraries and to 
monitor trends, they have not been adequate to assess the emerging uses of technology for access to 
information and the changing nature of research libraries. This is not to say that ARL had not been looking 
at other types of data or considering other possibilities for assessing library performance. As early as 1982, 
ARL contracted with Paul Kantor of Rutgers University to test four performance measures (availability of 
library materials, accessibility of library materials, analysis of patron activity, and delay analysis of 
interlibrary loan) and to produce a manual™ 1 to help libraries collect and analyze the data. The project 
achieved its goals of identifying the measures, but the members determined that the process for data 
collection was too burdensome for regular use. 

Also in 1982, ARL began to collect data in a supplementary statistics form that gave members a chance to 
test collection of specific data such as monographs purchased, staffing, monograph expenditures, staff 
salaries and fringe benefits, automation expenditures, external and reserve circulation, reference transactions, 
branch libraries, and instructional faculty. If variables were deemed useful, they were moved into the main 
ARL statistical survey. 

An emphasis in the need to collect measures on access arose from the Committee on ARL Statistics in 1989. 
In the early 1990s, an “Inventory of Library Access Characteristics” that covered library facilities, 
equipment, resource-sharing, and some access services was developed and conducted annually for several 
years. In an article for ARL’s newsletter, Sarah Pritchard outlined some of the efforts ARL had been 
undergoing to integrate measures of outputs and services and noted that a distinction between access 
measures, which were reflected by library services and could be collected at a national level, and 
performance measures, which spoke to the effective management and were better addressed locally, was 
deemed essential. 1 * 

ARL Statistics and Measurement Program 

In 1994, the Association adopted a new strategic objective that broadened the emphasis of the Statistics 
Program from just describing research libraries to one of “measuring the performance of research libraries 
and their contributions to teaching, research, scholarship and community service.” In conjunction with this 
objective, the membership supported an increase in dues allocation to the program and hired a full-time 
program officer. Since that time, the activities of the program expanded considerably. 

The current ARL Statistics and Measurement Program, 51 under the direction of the Statistics and 
Measurement Committee, provides descriptive data about the characteristics of research libraries today and 
seeks to develop and measure these libraries’ contributions. It has been instrumental in addressing the new 
ARL strategic goal of developing new measures for assessing library performance and their contribution to 
higher education. Reports and updates on activities are found in the annual ARL Program Plan 1 ARL’s 
bimonthly report/ 11 



The program supports the production of publications and member-distributed reports regarding the 
operations of research libraries. 3 " 111 In addition to the printed publishing efforts, the program has a strong 
presence in electronic publishing activities. Except for salary data, all data for the annual statistical 
publications are collected through a website interface to speed the data entry process and ensure accuracy. 

The ARL statistical survey instruments, through either use of data variables and definitions or in its entirety, 
has been used as the basis for surveys conducted by several other organizations: American Library 
Association (ALA) salary survey, Integrated Postsecondary Education Data System (IPEDS) Academic 
Library Survey; Association of College and Research Libraries (ACRL), a division of the American Library 
Association; Council on East Asian Libraries (CEAL); and the Canadian Association of Research Libraries 
(CARL). 



The New Measures Initiative 

The ARL New Measures Initiative was begun because of increasing demand for libraries to demonstrate 
outcomes and impacts in areas of importance to their institution and increasing pressure to maximize use of 
resources through benchmarking resulting in either cost savings or reallocation. Members felt that ARL was 
quite good at gathering data on inputs such as collection size, expenditures, and staffing and had made 
progress in the area of outputs through data on services and people served. And while some effort had been 
made to look at performance measures in the case of ratios (e.g., expenditures per FTE), a set of measures to 
determine outcomes or impacts had not yet been developed. 

In January 1999, several members of the Statistics and Measurement committee, the ARL Leadership and 
Management Committee, and other interested member leaders of ARL, gathered in a retreat setting to discuss 
what ARL can do to assist members in developing new measures that better describe research libraries and 
their services. Those attending the retreat addressed a set of questions regarding the data needed to describe 
research libraries in today’s environment, the need for new measures, and the means by which useful data 
and measurement tools could be developed. xlv The retreat participants recognized that any new measures 
must: 

• Be consistent with organizational missions, goals and objectives 

• Be integrated with an institution’s program review 

• Balance customer, stakeholder, and employee interests and needs 

• Establish accountability 

• Include the collection and use of reliable and valid dataThey determined that in order to succeed there 
must be collaboration among member leaders with strong interest in this area, specific projects 
developed with different models for exploration, and an intent to make resulting tools and methodologies 
available to full membership and wider community. Therefore, not all members are required to 
participate in all projects. This gives each project the flexibility it needs to test and refine measures 
without placing undue burdens on the entire ARL membership. 

When the New Measures Initiative first began, members examined the eight areas of interest generated at the 
1999 retreat and subsequently focused attention on several specific topics: higher education outcomes 
assessment, the utility of service effectiveness measures across libraries, usage measures for electronic 
resources, identification of cost drivers, and applying the results of the ILL/DD Performance Measures cost 
study. xv 

A series of specific projects were initiated in 2000, some supported with direct member financial 
contributions. The projects can be categorized as: 

• Demonstration project for service effectiveness measures 

• Investigation of the role libraries can play in campus learning outcomes activities 

• Project to define usage measures for electronic information resources 
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• Identification of cost-drivers and development of cost-benefit studies 

• Develop assisted self-study program to apply results of ILL/DD studies 

• Investigation of role libraries play in support of the research process 

This set of projects has now become the New Measures Initiative and incorporates initial investigations with 
a variety of projects at different stages of development. Two of the projects are large-scale. To keep the 
community informed about the initiative and the projects as they progress, a new measures website was 
established and organized by project. xvl 

LibQUAL+ 

The largest of the New Measures Initiatives to date is LibQUAL+ xvn , a research and development project 
undertaken by ARL in collaboration with Texas A&M University to define and measure library service 
quality across institutions and to create useful quality-assessment tools for local planning, such as the 
evaluation of a library’s collections-related services from the user’s point of view. LibQUAL+ was begun in 
1999 in response to members’ desire for alternative assessment methods. The project was spearheaded by 
Texas A&M University Libraries, who had been using a modified version of the SERVQUAL instrument--a 
customer survey used widely in the private sector-to evaluate their library services since the early 1990s. In 
the fall of 2000, ARL was awarded a US$498,368 grant from the U.S. Department of Education’s Fund for 
the Improvement of Post-Secondary Education (FIPSE) to help defray the cost of further developing the 
LibQUAL+ tool and scaling up its application to the full spectrum of libraries in the higher education 
community. The goals of the project include (a) the development of a regrounded protocol to evaluate 
service quality in all post-secondary libraries; (b) an effective web-based delivery mechanism for the 
protocol; (c) identification of best practices to allow wiser allocation of scarce resources through cross- 
institutional comparison; and (d) the establishment of an ongoing, cost-recovery, service quality assessment 
program at ARL. Receipt of the grant has allowed ARL to expand the project to libraries outside the ARL 
membership. 

As the LibQUAL+ dataset becomes richer, those libraries who participate in the survey and have users who 
rate services below minimum expectations can look to their cohorts who excel for models for improvement. 

E-Metrics 

First known as the e-Usage (Usage Measures for Electronic Resources) project, the ARL E-metrics project is 
an effort to explore the feasibility of collecting data on the usage of electronic resources. Twenty-four ARL 
member libraries committed US$10,000 to participate in a 20-month project to be carried out under contract 
with the Information Use Management and Policy Institute, School of Information Studies at Florida State 
University. 

Project goals are to (a) develop, test, and refine selected statistics and performance measures to describe 
electronic services and resources in ARL libraries; (b) engage in a collaborative effort with selected database 
vendors to establish an ongoing means to produce selected descriptive statistics on database use, users, and 
services; and (c) develop a proposal for external funding to maintain the development and refinement of 
networked statistics and performance measures. XVU1 

First completed was a knowledge inventory of ARL libraries with indications of institutions worth 
considering for best practices. Participating institutions subsequently worked with the project investigators 
to refine a set of measures - unfortunately not as small a list as many had hoped - to field test in spring 2001 
for which to develop tools and a methodology for data collection. Since many electronic measures are 
dependent on vendor data, a meeting with 12 database vendors (ones with which ARL libraries spend the 
most money) was held in March 2001 . The vendors also agreed to a field test of their data in collaboration 
with some of the participating libraries. The project investigators would review the data to see if a small set 
can be defined that are in line with library interests and can be generated by the vendors with some 
consistency. The investigators, project directors, and ARL staff have been engaged in discussions with other 
national and international organizations struggling with the same issues of electronic resource statistics, in 
particular vendor-based statistics, and a means for collaborating with those organizations while not 
detracting from or slowing down the ARL effort. 



Project investigators have also begun to consider development of an institutional outcomes model that can be 
applied to research libraries. Project participants will be responding to draft papers that outline either 
multiple models or possibly a process by which institutions develop their own outcomes in relation to 
institutional outcomes. 



Other Projects 

While the two projects are by far the largest, there are other projects that address the desire for ARL 
members to have new measures or new ways to measure their operations. To advance an investigation of the 
role libraries could play in addressing learning outcomes, Kenneth Smith, Eller Distinguished Service 
Professor of Economics at the University of Arizona was hired as a consultant to draft a white paper 
suggesting a role for research libraries. His white paper, “New Roles and Responsibilities for the University 
Library: Advancing Student Learning Through Outcomes Assessment” xlx outlines a strategy for involving 
research libraries in campus assessment activities to demonstrate the value of the library to the learning 
community. The paper was presented to the ARL membership at their May 2000 meeting and an action plan 
was developed and approved by the Statistics Committee to begin with a call for participants in summer 
2001. 

Doug Jones, University of Arizona, is serving as a Visiting Program Officer in 2001 to explore the impact 
libraries have on research and the research process. He will provide a report on his findings to the Statistics 
and Measurement Committee. If the results suggest future action, a project plan will be developed. 

Another project is the development of an assisted self-study for ILL/DD operations. It will consist of three 
parts: an organizational assessment, comparison of local activity against the benchmarks and best practices 
identified in the ILL/DD Performance Measures Study, and development of specific actions and changes that 
will result in a service that meets or exceeds those benchmarks. A small pilot group of libraries will test the 
methodology, with a goal of making the study available to the membership and library community in 2001 . 

Cost drivers continue to be of interest to some participating in the New Measures Initiative. Strategies to 
begin those projects include the development and review of a list of library functions to identify promising 
areas, clearly define processes that are a part of the function to investigate, develop common definition of 
tasks, conduct data gathering and analysis, and provide workshops and training on assessment methods. 

The first costs project is a technical services cost study methodology that is currently being tested at five 
research libraries.™ When the testing is complete and the software becomes generally available, the ARL 
Statistics and Measurement Committee will examine its use and consider it for possible recommendation to 
the ARL community. 

Eileen Hitchingham (Virginia Polytechnic and State University) developed a methodology to allocate staff 
costs to library services. Several libraries are testing the methodology in summer 2001 and, if successful and 
scalable, the methodology may subsequently be offered by ARL at a cost-recovery rate for members. 

The shift from an input to an outcomes focus requires a change in thinking for many individuals. In order to 
assist ARL member libraries to make this shift, the ARL Statistics and Measurement Program conducts a 
variety of workshops, sponsors conferences, and provides consulting services. 

Conclusion 

As the research libraries continue to feel the pressures of shifting from a management system that is 
accustomed to measures of inputs (revenues) and outputs (expenditures) to one of efficiency and 
effectiveness, it will be important that the New Measures Initiative keep looking for innovative ways to 
describe the research libraries of today and their contributions to their organizations. Whether some or all of 
the ARL members choose to participate in some or all of the projects is not as important as offering the 
opportunity to test and refine the new measures. As individual New Measures projects develop, there will 
continue to be changes in the activities and it is likely each year will bring modifications in the agenda. And 
as the stages of the various projects in the New Measures Initiative are completed, the ARL Statistics and 
Measurement Program, in conjunction with the Statistics Committee, will determine how best to deploy the 
results of the project or how to take the project further. Likely scenarios for many of the projects include the 



incorporation of data elements into the statistical surveys, the development of workshops and consulting 
services for performance measures, and the establishment of data gathering and statistical analysis tools that 
the ARL Statistics and Measurement Program can offer on a cost-recovery basis. 
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